An Information-Theoretic External Cluster-Validity Measure

In this paper we propose a measure of sim­ ilarity /association between two partitions of a set of objects. Our motivation is the desire to use the measure to characterize the quality or accuracy of clustering algorithms by some­ how comparing the clusters they produce with "ground truth" consisting of classes as­ signed by manual means or some other means in whose veracity there is confidence. Such measures are referred to as "external". Our measure also allows clusterings with differ­ ent numbers of clusters to be compared in a quantitative and principled way. Our eval­ uation scheme quantitatively measures how useful the cluster labels are as predictors of their class labels. It computes the reduction in the number of bits that would be required to encode (compress) the class labels if both the encoder and decoder have free access to the cluster labels. To achieve this encoding the estimated conditional probabilities of the class labels given the cluster labels must also be encoded. In addition to defining the mea­ sure we compare it to other commonly used external measures and demonstrate its supe­ riority as judged by certain criteria. 1 The Clustering Problem The most common unsupervised-learning problem is clustering, in which we are given a set 0 of objects or patterns 0 = {w; li = 1, 2, . . . , n} and each object has a representation x; := x(w; ) in some feature space which is frequently treated as an m-dimensional continuum !Rm. Some of the features may be categorical, however. The goal in clustering is to group the objects by grouping their associated feature vectors X = { x;li = 1, 2, . .. , n} . This grouping can be based on any num­ ber of criteria. It is assumed that the dimensions of x are attributes relevant to some application of interest. The grouping is performed on the basis of some mea­ sure of similarity relevant to the application and as­ sociated feature space. There are numerous objective functions and algorithms for clustering (see (JD88] for a survey), but we are not concerned with these here. Our task is to devise a measure of the quality of the output of clustering algorithms. Let K = {k;li = 1, 2, . . . , n} be a set of cluster labels assigned to the elements of X. The labels themselves are taken from a set K., where IKI is the number of clusters. We have some clustering procedure f that maps X to K. Definition: clustering procedure f: f : X(O) -+ K(O) (1) The procedure f may determine the optimal number of clusters as well as the assignment of feature vectors (objects) to class labels or it may accept the number of clusters as input. The set 0 can be considered to have been drawn from some larger population, which can be characterized by a probability density p( x). The combination of p( x) and the clustering procedure f results in a probability distribution {p( k)} over cluster labels. We define three clustering problems: (1) Each pat­ tern is assigned to one and only one cluster so-called partitional clustering. (2) Each pattern may be as­ signed to multiple clusters. These are binary assign­ ments. ( 3) Each pattern has a degree of membership in each cluster. The measure we propose applies to parti­ tional clustering. In addition to these three categories a distinction can be made between flat and hierarchical clustering (although a flat is technically a special case of hierarchicali.e. a depth-one tree). Our measure applies to flat clustering.